
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Units test facility &#8212; Universal Ctags 0.3.0 documentation</title>
    <link rel="stylesheet" href="_static/classic.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '0.3.0',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true,
        SOURCELINK_SUFFIX: '.txt'
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Semi-fuzz(Fuzz) testing" href="semifuzz.html" />
    <link rel="prev" title="Testing ctags" href="testing.html" /> 
  </head>
  <body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="semifuzz.html" title="Semi-fuzz(Fuzz) testing"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="testing.html" title="Testing ctags"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> &#187;</li>
          <li class="nav-item nav-item-1"><a href="testing.html" accesskey="U">Testing ctags</a> &#187;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="units-test-facility">
<h1><em>Units</em> test facility<a class="headerlink" href="#units-test-facility" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Maintainer:</th><td class="field-body">Masatake YAMATO &lt;<a class="reference external" href="mailto:yamato&#37;&#52;&#48;redhat&#46;com">yamato<span>&#64;</span>redhat<span>&#46;</span>com</a>&gt;</td>
</tr>
</tbody>
</table>
<hr class="docutils" />
<p>Exuberant ctags has a test facility. The test case were <em>Test</em>
directory. So Here I call it <em>Test</em>.</p>
<p>Main aim of the facility is detecting regression. All files under Test
directory are given as input for old and new version of ctags
commands.  The output tags files of both versions are compared. If any
difference is found the check fails. <em>Test</em> expects the older ctags
binary to be correct.</p>
<p>This expectation is not always met. Consider that a parser for a new
language is added. You may want to add a sample source code for that
language to <em>Test</em>. An older ctags version is unable to generate a
tags file for that sample code, but the newer ctags version does. At
this point a difference is found and <em>Test</em> reports failure.</p>
<p>The units test facility(<em>Units</em>) I describe here takes a different
approach. An input file and an expected output file are given by a
contributor of a language parser. The units test facility runs ctags
command with the input file and compares its output and the expected
output file. The expected output doesn’t depend on ctags.</p>
<p>If a contributor sends a patch which may improve a language parser,
and if a reviewer is not familiar with that language, s/he cannot
evaluate it.</p>
<p><em>Unit</em> test files, the pair of input file and expected output file may
be able to explain the intent of patch well; and may help the
reviewer.</p>
<div class="section" id="how-to-write-a-test-case">
<h2>How to write a test case<a class="headerlink" href="#how-to-write-a-test-case" title="Permalink to this headline">¶</a></h2>
<p>The test facility recognizes an input file and an expected
output file by patterns of file name. Each test case should
have its own directory under Units directory.</p>
<p><em>Units/TEST/input.*</em> <strong>requisite</strong></p>
<blockquote>
<div>Input file name must have a <em>input</em> as basename. <em>TEST</em>
part should explain the test case well.</div></blockquote>
<p><em>Units/TEST/input-[0-9].*</em> <em>Units/TEST/input-[0-9]-</em>.** <strong>optional</strong></p>
<blockquote>
<div>Optional input file names. They are put next to <em>input.*</em> in
testing command line.</div></blockquote>
<p><em>Units/TEST/expected.tags</em> <strong>optional</strong></p>
<blockquote>
<div><p>Expected output file must have a name <em>expected.tags</em>. It
should be the same directory of the input file.</p>
<p>If this file is not given, the exit status of ctags process
is just checked; the output is ignored.</p>
<p>If you want to test etags output (specified with <code class="docutils literal"><span class="pre">-e</span></code> ),
Use <strong>.tags-e</strong> as suffix instead of <strong>.tags</strong>.
In such case you don’t have to write <code class="docutils literal"><span class="pre">-e</span></code> to <code class="docutils literal"><span class="pre">args.ctags</span></code>.
The test facility sets <code class="docutils literal"><span class="pre">-e</span></code> automatically.</p>
<p>If you want to test cross reference output (specified with <code class="docutils literal"><span class="pre">-x</span></code> ),
Use <strong>.tags-x</strong> as suffix instead of <strong>.tags</strong>.
In such case you don’t have to write <code class="docutils literal"><span class="pre">-x</span></code> to <code class="docutils literal"><span class="pre">args.ctags</span></code>.
The test facility sets <code class="docutils literal"><span class="pre">-x</span></code> automatically.</p>
<p>If you want to test json output (specified with <code class="docutils literal"><span class="pre">--output-format=json</span></code> ),
Use <strong>.tags-json</strong> as suffix instead of <strong>.tags</strong>.
In such case you don’t have to write <code class="docutils literal"><span class="pre">--output-format=json</span></code> to <code class="docutils literal"><span class="pre">args.ctags</span></code>,
and <code class="docutils literal"><span class="pre">json</span></code> to <code class="docutils literal"><span class="pre">features</span></code>.
The test facility sets the option and the feature automatically.</p>
</div></blockquote>
<p><em>Units/TEST/args.ctags</em> <strong>optional</strong></p>
<blockquote>
<div><p><code class="docutils literal"><span class="pre">-o</span> <span class="pre">-</span></code> is used as default optional argument when running a
unit test ctags. If you want to add more options, enumerate
options in <strong>args.ctags</strong> file. This file is an optional.</p>
<p>Remember you have to put one option in one line; don’t
put multiple options to one line. Multiple options in
one line doesn’t work.</p>
</div></blockquote>
<p><em>Units/TEST/filter-*.*</em> <strong>optional</strong></p>
<blockquote>
<div>You can rearrange the output of ctags with this command
before comparing with <em>executed.tags</em>.
This command is invoked with no argument. The output
ctags is given via stdin. Rearrange data should be
written to stdout.</div></blockquote>
<p><em>Units/TEST/features</em> <strong>optional</strong></p>
<blockquote>
<div><p>If a unit test case requires special features of ctags,
enumerate them in this file line by line. If a target ctags
doesn’t have one of the features, the test is skipped.</p>
<p>If a file line is started with <code class="docutils literal"><span class="pre">!</span></code>, the effect is inverted;
if a target ctags has the feature specified with <code class="docutils literal"><span class="pre">!</span></code>, the
test is skipped.</p>
<p>All features built-in can be listed with passing
<code class="docutils literal"><span class="pre">--list-features</span></code> to ctags.</p>
</div></blockquote>
<p><em>Units/TEST/languages</em> <strong>optional</strong></p>
<blockquote>
<div><p>If a unit test case requires that language parsers are enabled/available,
enumerate them in this file line by line. If one of them is
disabled/unavailable, the test is skipped.</p>
<p>language parsers enabled/available can be checked with passing
<code class="docutils literal"><span class="pre">--list-languages</span></code> to ctags.</p>
</div></blockquote>
<p><em>Units/TEST/dictfile</em> <strong>optional</strong></p>
<blockquote>
<div>Used in spell checking.
See <a class="reference internal" href="cspell.html#cspell"><span class="std std-ref">cspell</span></a> for more details.</div></blockquote>
</div>
<div class="section" id="note-for-importing-a-test-case-from-test-directory">
<h2>Note for importing a test case from Test directory<a class="headerlink" href="#note-for-importing-a-test-case-from-test-directory" title="Permalink to this headline">¶</a></h2>
<p>I think all test cases under Test directory should be converted to
Units.</p>
<p>If you convert use following TEST name convention.</p>
<ul class="simple">
<li>use <em>.t</em> instead of <em>.d</em> as suffix for the name</li>
</ul>
<p>Here is an example:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Test</span><span class="o">/</span><span class="n">simple</span><span class="o">.</span><span class="n">sh</span>
</pre></div>
</div>
<p>This should be:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Units</span><span class="o">/</span><span class="n">simple</span><span class="o">.</span><span class="n">sh</span><span class="o">.</span><span class="n">t</span>
</pre></div>
</div>
<p>With this name convention we can track which test case is converted or
not.</p>
</div>
<div class="section" id="example-of-files">
<h2>Example of files<a class="headerlink" href="#example-of-files" title="Permalink to this headline">¶</a></h2>
<p>See <em>Units/c-sample/input.c</em> and <em>Units/c-sample/expected</em>.</p>
</div>
<div class="section" id="how-to-run-unit-tests">
<h2>How to run unit tests<a class="headerlink" href="#how-to-run-unit-tests" title="Permalink to this headline">¶</a></h2>
<p><em>test</em> make target:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units
</pre></div>
</div>
<p>The result of unit tests is reported by lines. You can specify
test cases with <code class="docutils literal"><span class="pre">UNITS=</span></code>.</p>
<p>An example to run <em>vim-command.d</em> only:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units UNITS=vim-command
</pre></div>
</div>
<p>Another example to run <em>vim-command.d</em> and <em>parser-python.r/bug1856363.py.d</em>:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units UNITS=vim-command,bug1856363.py
</pre></div>
</div>
<p>During testing <em>OUTPUT.tmp</em>, <em>EXPECTED.tmp</em> and <em>DIFF.tmp</em> files are
generated for each test case directory. These are removed when the
unit test is <strong>passed</strong>.  If the result is <strong>FAILED</strong>, it is kept for
debugging. Following command line can clean up these generated files
at once:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make clean-units
</pre></div>
</div>
<p>Other than <strong>FAILED</strong> and <strong>passed</strong> two types of result are
defined.</p>
<p><strong>skipped</strong></p>
<blockquote>
<div>means running the test case is skipped in some reason.</div></blockquote>
<p><strong>failed (KNOWN bug)</strong></p>
<blockquote>
<div>mean the result if failed but the failure is expected.
See “Gathering test cases for known bugs”.</div></blockquote>
</div>
<div class="section" id="example-of-running">
<h2>Example of running<a class="headerlink" href="#example-of-running" title="Permalink to this headline">¶</a></h2>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units
Category: ROOT
-------------------------------------------------------------------------
Testing 1795612.js as JavaScript                            passed
Testing 1850914.js as JavaScript                            passed
Testing 1878155.js as JavaScript                            passed
Testing 1880687.js as JavaScript                            passed
Testing 2023624.js as JavaScript                            passed
Testing 3184782.sql as SQL                                  passed
...
</pre></div>
</div>
</div>
<div class="section" id="running-unit-tests-for-specific-languages">
<h2>Running unit tests for specific languages<a class="headerlink" href="#running-unit-tests-for-specific-languages" title="Permalink to this headline">¶</a></h2>
<p>You can run only the tests for specific languages by setting
<code class="docutils literal"><span class="pre">LANGUAGES</span></code> to parsers as reported by
<code class="docutils literal"><span class="pre">ctags</span> <span class="pre">--list-languages</span></code>:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">make</span> <span class="n">units</span> <span class="n">LANGUAGES</span><span class="o">=</span><span class="n">PHP</span><span class="p">,</span><span class="n">C</span>
</pre></div>
</div>
<p>Multiple languages can be selected using a comma separated list.</p>
</div>
<div class="section" id="gathering-test-cases-for-known-bugs">
<h2>Gathering test cases for known bugs<a class="headerlink" href="#gathering-test-cases-for-known-bugs" title="Permalink to this headline">¶</a></h2>
<p>When we met a bug, making a small test case that triggers the bug is
important development activity. Even the bug cannot be fixed in soon,
the test case is an important result of work. Such result should
be merged to the source tree. However, we don’t love <strong>FAILED</strong>
message, too. What we should do?</p>
<p>In such case, merge as usually but use <em>.b</em> as suffix for
the directory of test case instead of <em>.d</em>.</p>
<p><em>Unix/css-singlequote-in-comment-issue2.b</em> is an example
of <em>.b</em> suffix usage.</p>
<p>When you run test.units target, you will see:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Testing</span> <span class="n">c</span><span class="o">-</span><span class="n">sample</span> <span class="k">as</span> <span class="n">C</span>                                 <span class="n">passed</span>
<span class="n">Testing</span> <span class="n">css</span><span class="o">-</span><span class="n">singlequote</span><span class="o">-</span><span class="ow">in</span><span class="o">-</span><span class="n">comment</span> <span class="k">as</span> <span class="n">CSS</span>             <span class="n">failed</span> <span class="p">(</span><span class="n">KNOWN</span> <span class="n">bug</span><span class="p">)</span>
<span class="n">Testing</span> <span class="n">ctags</span><span class="o">-</span><span class="n">simple</span> <span class="k">as</span> <span class="n">ctags</span>                         <span class="n">passed</span>
</pre></div>
</div>
<p>Suffix <em>.i</em> is a variant of <em>.b</em>. <em>.i</em> is for merging/gathering input
which lets ctags process enter an infinite loop. Different from <em>.b</em>,
test cases marked as <em>.i</em> are never executed. They are just skipped
but reported the skips:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">Testing</span> <span class="n">ada</span><span class="o">-</span><span class="n">ads</span> <span class="k">as</span> <span class="n">Ada</span>                                <span class="n">passed</span>
<span class="n">Testing</span> <span class="n">ada</span><span class="o">-</span><span class="n">function</span> <span class="k">as</span> <span class="n">Ada</span>                           <span class="n">skipped</span> <span class="p">(</span><span class="n">may</span> <span class="n">cause</span> <span class="n">an</span> <span class="n">infinite</span> <span class="n">loop</span><span class="p">)</span>
<span class="n">Testing</span> <span class="n">ada</span><span class="o">-</span><span class="n">protected</span> <span class="k">as</span> <span class="n">Ada</span>                          <span class="n">passed</span>
<span class="o">...</span>

<span class="n">Summary</span> <span class="p">(</span><span class="n">see</span> <span class="n">CMDLINE</span><span class="o">.</span><span class="n">tmp</span> <span class="n">to</span> <span class="n">reproduce</span> <span class="n">without</span> <span class="n">test</span> <span class="n">harness</span><span class="p">)</span>
<span class="o">------------------------------------------------------------</span>
  <span class="c1">#passed:                                347</span>
  <span class="c1">#FIXED:                                 0</span>
  <span class="c1">#FAILED (unexpected-exit-status):       0</span>
  <span class="c1">#FAILED (unexpected-output):            0</span>
  <span class="c1">#skipped (features):                    0</span>
  <span class="c1">#skipped (languages):                   0</span>
  <span class="c1">#skipped (infinite-loop):               1</span>
    <span class="n">ada</span><span class="o">-</span><span class="n">protected</span>
  <span class="o">...</span>
</pre></div>
</div>
</div>
<div class="section" id="running-under-valgrind-and-timeout">
<h2>Running under valgrind and timeout<a class="headerlink" href="#running-under-valgrind-and-timeout" title="Permalink to this headline">¶</a></h2>
<p>If <code class="docutils literal"><span class="pre">VG=1</span></code> is given, each test cases are run under valgrind.
If valgrind detects an error, it is reported as:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units VG=1
Testing css-singlequote-in-comment as CSS             failed (valgrind-error)
...
Summary (see CMDLINE.tmp to reproduce without test harness)
------------------------------------------------------------
...
#valgrind-error:                        1
  css-singlequote-in-comment
...
</pre></div>
</div>
<p>In this case the report of valgrind is recorded to
<code class="docutils literal"><span class="pre">Units/css-singlequote-in-comment/VALGRIND-CSS.tmp</span></code>.</p>
<p>NOTE: <code class="docutils literal"><span class="pre">/bin/bash</span></code> is needed to report the result. You can specify a shell
running test with SHELL macro like:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units VG=1 SHELL=/bin/bash
</pre></div>
</div>
<p>If <code class="docutils literal"><span class="pre">TIMEOUT=N</span></code> is given, each test cases are run under timeout
command. If ctags doesn’t stop in <code class="docutils literal"><span class="pre">N</span></code> second, it is stopped
by timeout command and reported as:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units TIMEOUT=1
Testing css-singlequote-in-comment as CSS             failed (TIMED OUT)
...
Summary (see CMDLINE.tmp to reproduce without test harness)
------------------------------------------------------------
...
#TIMED-OUT:                             1
  css-singlequote-in-comment
...
</pre></div>
</div>
<p>If <code class="docutils literal"><span class="pre">TIMEOUT=N</span></code> is given, <em>.i</em> test cases are run. They will be
reported as <em>TIMED-OUT</em>.</p>
</div>
<div class="section" id="categories">
<h2>Categories<a class="headerlink" href="#categories" title="Permalink to this headline">¶</a></h2>
<p>With <em>.r</em> suffix, you can put test cases under a sub directory
of <em>Units</em>. <code class="docutils literal"><span class="pre">Units/parser-ada.r</span></code> is an example. If <em>misc/units</em>
test harness, the sub directory is called a category. <code class="docutils literal"><span class="pre">parser-ada.r</span></code>
is the name category in the above example.</p>
<p><em>CATEGORIES</em> macro of make is for running units in specified categories.
Following command line is for running units in
<code class="docutils literal"><span class="pre">Units/parser-sh.r</span></code> and <code class="docutils literal"><span class="pre">Units/parser-ada.r</span></code>:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span>$ make units CATEGORIES=&#39;parser-sh,parser-ada&#39;
</pre></div>
</div>
</div>
<div class="section" id="finding-minimal-bad-input">
<h2>Finding minimal bad input<a class="headerlink" href="#finding-minimal-bad-input" title="Permalink to this headline">¶</a></h2>
<p>When a test case is failed, the input causing <code class="docutils literal"><span class="pre">FAILED</span></code> result is
passed to <em>misc/units shrink</em>.  <em>misc/units shrink</em> tries to make the
shortest input which makes ctags exits with non-zero status.  The
result is reported to <code class="docutils literal"><span class="pre">Units/\*/SHRINK-${language}.tmp</span></code>.  Maybe
useful to debug.</p>
</div>
<div class="section" id="acknowledgments">
<h2>Acknowledgments<a class="headerlink" href="#acknowledgments" title="Permalink to this headline">¶</a></h2>
<p>The file name rule is suggested by Maxime Coste &lt;<a class="reference external" href="mailto:frrrwww&#37;&#52;&#48;gmail&#46;com">frrrwww<span>&#64;</span>gmail<span>&#46;</span>com</a>&gt;.</p>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#"><em>Units</em> test facility</a><ul>
<li><a class="reference internal" href="#how-to-write-a-test-case">How to write a test case</a></li>
<li><a class="reference internal" href="#note-for-importing-a-test-case-from-test-directory">Note for importing a test case from Test directory</a></li>
<li><a class="reference internal" href="#example-of-files">Example of files</a></li>
<li><a class="reference internal" href="#how-to-run-unit-tests">How to run unit tests</a></li>
<li><a class="reference internal" href="#example-of-running">Example of running</a></li>
<li><a class="reference internal" href="#running-unit-tests-for-specific-languages">Running unit tests for specific languages</a></li>
<li><a class="reference internal" href="#gathering-test-cases-for-known-bugs">Gathering test cases for known bugs</a></li>
<li><a class="reference internal" href="#running-under-valgrind-and-timeout">Running under valgrind and timeout</a></li>
<li><a class="reference internal" href="#categories">Categories</a></li>
<li><a class="reference internal" href="#finding-minimal-bad-input">Finding minimal bad input</a></li>
<li><a class="reference internal" href="#acknowledgments">Acknowledgments</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="testing.html"
                        title="previous chapter">Testing ctags</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="semifuzz.html"
                        title="next chapter">Semi-fuzz(<em>Fuzz</em>) testing</a></p>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <div><input type="text" name="q" /></div>
      <div><input type="submit" value="Go" /></div>
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="semifuzz.html" title="Semi-fuzz(Fuzz) testing"
             >next</a> |</li>
        <li class="right" >
          <a href="testing.html" title="Testing ctags"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Universal Ctags 0.3.0 documentation</a> &#187;</li>
          <li class="nav-item nav-item-1"><a href="testing.html" >Testing ctags</a> &#187;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2015, Universal Ctags Team.
      Last updated on 19 Dec 2017.
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.4.
    </div>
  </body>
</html>